Methods and apparatuses consistent with the present invention relate to informing at least one occupant of a vehicle, such as a motor vehicle, of the surroundings of the vehicle. The recording means of the vehicle generate at least two images, which contain different road sections of the road plane in the surroundings of the vehicle, and the at least two images are processed by an image processing unit and are displayed on a display unit in the interior of the vehicle.
Methods that are intended for informing the driver and that display a plurality of images of road sections in the vehicle's surroundings on a display unit are known from the related art. The document DE 10 2006 036 933 A1 shows such a method, wherein a plurality of individual images of the vehicle's surroundings that are recorded by the vehicle's cameras are suitably transformed in order to generate a composite picture that reproduces on a display unit, in a bird's eye perspective, the contour of the vehicle and suitably assembled individual images of the vehicle's surroundings.
The document DE 10 2006 007 493 A1 discloses a device for seeing the periphery of a vehicle, wherein an imaging camera captures the horizontal front region in the front region of the vehicle and also a vertical region in front of and behind the vehicle. The regions are assembled into a panorama image of the front surroundings of the vehicle in a display unit. In this process the vehicle region in front of the vehicle is distorted in a convex manner, in order to generate a suitable perspective with the use of this method.
The related art methods for informing the driver have the drawback that the camera often records a number of images that contain different regions of the surroundings of the vehicle, but these images cannot be suitably displayed in a single depiction on a display unit in the vehicle interior. Especially for the purpose of recording the immediate near field, it is necessary to have a plurality of cameras owing to the shape of the vehicle. It is often the case that the user cannot correctly assign the individual images to the real surroundings. It is difficult for the user to understand the actual positions of the objects that are imaged in one or more images.
Therefore, an object of the invention is to provide a method, system, and/or vehicle for informing the at least one vehicle occupant of the surroundings of the vehicle by generating a depiction of different regions in the vehicle's surroundings in such a way that the depiction is easy for the occupant to understand.
According to embodiments of the invention, the recording means of the vehicle generates at least two images, which contain different road sections of the road plane in the surroundings of the vehicle. By “different road sections of the road plane” is meant those road sections that are generated, in particular, with a variety of recording means that are provided at different installation sites in the vehicle and/or correspond to various views relative to the vehicle's orientation, such as from the top, from the side, and/or rearwards. It involves, in particular, regions of the vehicle's surroundings that depict the parts of the road and, if desired, the objects on said road in such a way that the parts of the road do not overlap at least in a subarea (do not overlap at all or overlap just slightly).
By “recording means” is meant one or more devices that depict in graphic form a segment of the vehicle's surroundings for the driver of the vehicle and are technically capable of implementing the features of embodiments of the invention. Devices that can provide a segment of the vehicle's surroundings in the form of an image that can be shown to the driver are subsumed under the term “imaging device” within the scope of this application. Imaging devices in this context include both those that are based on camera images (including, for example, images recorded in the infrared wavelength range) and also those that are based on synthetic images generated by means of a computer (for example, from sensor data). These images can be generated, for example, by scanning the surroundings by means of a radar device, a lidar device, ultrasonic sensors and/or clusters of such sensors or similar recording means. In particular, the images can contain both a portion of the vehicle's surroundings and also artificially generated graphics, which are synthesized, for example, by means of sensor data and/or symbolic graphics.
Even such cases in which images are generated in real time on the basis of scanning the surroundings are included under the concept “generation of at least two images with the use of recording means” that is used herein. An imaging device in the above sense can also include a plurality of cameras and/or other sensors, of which the images are stitched together to form a single image. By the term “image” is meant not only a static image representing a single recording of the surroundings, but also a dynamically changing image in the form of a video stream composed of consecutive images, for example, in the form of an MPEG video stream or a sequence of complete images.
In the method according to embodiments of the invention, the at least two images are processed by means of an image processing unit and displayed on a display unit in the vehicle interior. In this context the display unit can comprise any device with which the images can be visually displayed, such as flat screen displays, heads-up displays, 3D displays, and (3D) projection devices. In a preferred variant the display unit is a central vehicle display. The term “vehicle” can be defined, for example, as a motor vehicle, an airplane, a watercraft or amphibian craft. Correspondingly the term “road section” can also be defined, for example, as the water surface or portions of a landing strip.
The method according to embodiments of the invention is characterized in that a depiction comprising a virtual road plane is generated. In this case at least two images are imaged on the virtual road plane, and the virtual road plane is displayed in perspective, wherein the perspective display is generated preferably by means of the limiting shape of the virtual road plane and/or the arrangement of at least two images in relation to each other.
In this context the term “virtual road plane” is defined in such a way that the perspective of the virtual road plane does not match the perspective of the recording means and, in particular, does not originate from the content of the images, but rather is generated only by means of the method according to embodiments of the invention, in particular by means of a depiction of the virtual road plane in a suitable form and arrangement of the images in relation to each other. The at least two transformed images are imaged essentially on a virtual road plane. In other words the virtual road plane is depicted as a plane that is tilted in perspective. The perspective of said plane does not match the perspective of the recording means. In this case the virtual road plane communicates to the at least one occupant of the vehicle the principle direction of the road that is seen from a specific perspective.
Working on this basis, the term “perspective display of the virtual road plane” is defined in such a way that the virtual road plane is depicted three-dimensionally or quasi-three-dimensionally in a specific perspective or a perspective that is determined according to specific criteria, so that the viewer has a three-dimensional spatial impression, especially with respect to the spatial orientation of the virtual road plane. In particular, the perspective depictions include such depictions with which the parts of a virtual plane are perceived as lying at different depths in relation to the display area of the display unit and/or are differently oriented in relation to said display area. Thus, a perspective display in this sense does not include a display without a spatial impression, such as depictions in a top view.
The method according to embodiments of the invention has the advantage that the arrangement and the position of the individual road sections in the surroundings of the vehicle can be understood in a fast and intuitive way by the driver by means of a quasi-three-dimensional depiction. In this case a depiction that is especially comprehensible is one in which the perspective of the displayed virtual road plane corresponds essentially to the orientation of the actual road plane, which could be seen by the driver of the vehicle if his vehicle were transparent, for example, towards the side.
The depiction of a virtual road that corresponds to the user's perception running diagonally downwards to the front, especially in a perceived viewing angle range between 30° and 60° between the viewing direction and the virtual road plane, is especially advantageous. In addition, this feature permits very good use of the display area of a conventionally shaped, horizontally extended vehicle display. The depiction allows an adequately detailed imaging of the regions that otherwise could not be displayed, for example, on a conventional vehicle screen exhibiting a side ratio of 9:16, without being unnaturally shrunk.
In this context the perspective display can be enhanced by means of an additional reference figure, for example, a frame and/or an additionally generated and depicted 3D action graphics imaging, which is displayed by the display unit. The reference figure can be constructed in such a way that it identifies the expansion and/or orientation of the virtual space. In this way the reference figure helps the user with the orientation and reinforces the spatial perception of the rest of the depiction.
It is especially advantageous if at least two images, which essentially capture a road plane and/or objects located on a road plane from a variety of installation sites on the vehicle and/or which are oriented differently in relation to the coordinate system of the vehicle, are imaged on this virtual road plane or portions thereof. In this case it is advantageous for the user if the virtual road plane has a different perspective orientation than the recording means. This feature allows the user to interpret the images correctly.
It is especially advantageous if the at least two images are imaged on the virtual road plane by means of a geometric transformation, which is applied in such a way that the transformation of the at least two images corresponds essentially to the perspective in which the virtual road plane is depicted. In this process the image can be formed by means of an imaging function determined by the method of embodiments of the invention, and/or by means of an allocation table for the individual image regions and/or pixels, and/or by means of methods that are known from the related art. The allocation table can determine the shift of certain representative points of the original image to the corresponding points of the transformed image. The points of the imaging that lie in-between can be determined by means of a position determined by an interpolation process. The transformation is designed preferably in such a manner that the distances between the points of the transformed images are in such a ratio to each other as they would be seen from a certain viewing point.
The delimiting form of the virtual imaging plane and/or the delimiting forms of individual images imaged thereon can take place with the use of a so-called mask, for example, through the selection of a pixel region of the transformed image that is imaged on the virtual plane. As an alternative or in addition, the delimiting form can also be generated by a selection of the image regions from the whole region, which can recorded by the recording means. The delimiting form can be depicted, for example, by means of a sudden or soft transition between the imaging of the transformed image and the background, and can be additionally reinforced by depicting a graphical frame for the virtual plane and/or for the individual transformed images imaged thereon.
A transformation of the images that is especially advantageous is a transformation in which the change in size of the images of the road sections in the event of a moving vehicle corresponds approximately to the perceived change in the size of these road sections that is perceived by a person looking directly at these road sections from a similar perspective and movement. Thus, when the vehicle is in motion (for example, during a parking maneuver), the images of the textures of the road (for example, paving stones or objects lying on the road) behave with respect to the viewer in the interior of the vehicle approximately as if the driver were to see them through the side regions of his vehicle.
Especially when the vehicle is in motion, the method according to embodiments of the invention offers an advantage over the related art flat top view depiction, where the imaging of the side regions of the vehicle is rotated by 90° in relation to the direction of travel.
The method according to embodiments of the invention is especially advantageous if the at least two images contain separate road sections without mutual overlapping.
Preferably the at least two images contain in each case a road section laterally next to the vehicle and/or behind the vehicle and/or in front of the vehicle. In this case the road sections are recorded with suitable imaging devices and are displayed as a single composite image on the display unit. This process provides the option of displaying in a simple and intuitive way the whole surrounding area around the vehicle in a three-dimensional or quasi-three-dimensional depiction on the display unit.
Preferably the control of the image parameters of the images from a variety of recording means (for example, the exposure control, the histogram operations, and/or the white balance) is designed in such a way that at least one image parameter of an image from the recording means depends on the scene parameters of a different image of another recording means.
In particular, the images from different recording means can be subjected to such a histogram operation that the major integral features of their histograms, especially luminosity and/or RGB histograms, are essentially adjusted to each other. Thus, for example, such a brightness/contrast and/or white balance control is reached that enables an essentially uniform perception of the color temperature of the individual images in the whole depiction. In this way it is possible to achieve, in particular, a conformity with respect to the human perception of these images, producing a state that makes it easier to interpret the images and additionally offers an aesthetic advantage.
Preferably the images from the different recording means are synchronized among each other, especially by the control and/or cycling operations from the image processing unit. In this way it is possible to achieve a conformal imaging of the objects on the road that expands, for example, from one image to another, especially when the vehicle is moving.
In an additional, especially preferred embodiment of the method according to the invention, the generated depiction is constructed in such a way that the positions of the parts of the vehicle's surroundings can be seen in relation to the vehicle and/or to specific parts of the vehicle. By this procedure the driver recognizes immediately by looking at the display unit, at which points in the surroundings of his vehicle the depicted road sections are located, without having to perform an analytic allocation of the individual road sections, for example, by means of his knowledge of the installation sites of the individual recording means on the vehicle and their orientation. The depiction can be provided with a superimposed scale and/or markings, preferably equidistant markings, which are perceived to be located above the virtual road plane preferably in the perception of the occupant of the vehicle.
In another, especially preferred embodiment of the method according to the invention, the generated depiction contains a vehicle model. In this case the perspective in which the vehicle model is depicted corresponds essentially to the perspective of the virtual road plane, wherein both perspectives correspond preferably to the same virtual viewing point, and the position of the vehicle model relative to the virtual road plane corresponds essentially to the actual position of the vehicle relative to the road. Preferably the imaging factor of the road sections imaged on the road plane coincides with the imaging factor of at least the bottom part of the vehicle with respect to its length in the direction of travel of the vehicle. In this way the occupant of the vehicle can always recognize the perspective in which the vehicle's surroundings are depicted by the method of embodiments of the invention, by means of the perspective of the depicted vehicle model.
The vehicle model can also be constructed as three-dimensional action graphics that depict the actual vehicle. In this case the spatial orientation of the model can be recognized, in particular, by the position of the outer edges of the model and/or by the imaging of typical design elements of the vehicle. The perspective imaging of the vehicle model does not contain a top view or a frontal view of the vehicle from the rear or from the front. The vehicle model can contain highly accentuated edges, in particular outer edges, preferably such edges that enhance the three-dimensional effect of the graphics. The vehicle model can display at least symbolically the special properties of the vehicle, such as color, model-specific features, etc.
The virtual road plane can be complemented in an advantageous manner with at least one additional virtual plane on which a part of the vehicle's surroundings is imaged and which is depicted as standing perpendicularly especially to the virtual road plane. To this end the imaged part of the vehicle's surroundings is captured with recording means on the vehicle and can contain any region of the vehicle's surroundings. The imaging of the part of the vehicle's surroundings on the at least one additional virtual plane takes place once more preferably by means of a geometric transformation, which is applied in such a way that the transformation of the part of the vehicle's surroundings corresponds essentially to the perspective in which the at least one additional virtual plane is depicted. Preferably regions of the vehicle's surroundings that are captured in essence from a different perspective than that of the at least two images, which are displayed in the virtual road plane, are depicted on these additional virtual planes. These images can come, in particular, from recording means that are installed at a different point on the vehicle. By this process a kind of virtual space is constructed for the viewer that depicts the surroundings of the vehicle by means of a plurality of perspectives, where the depiction mode contains the information about the respective perspectives.
Preferably the image(s) displayed in the at least one additional virtual plane contain the front and/or rear and/or side surroundings of the vehicle captured in perspective.
Preferably the reference of the virtual planes (the at least one additional virtual plane and/or the virtual road plane) the points of the vehicle where the recording means are located is highlighted in graphic form. This feature can be achieved, in particular, by positioning the virtual planes inside the depiction, especially relative to the vehicle model. As an alternative or in addition, this goal can be achieved, for example, with connecting lines that are superimposed on the depiction and produce a connection between the parts of the vehicle model and the virtual planes, in particular, their corner points. In this way even a new user can be absolutely sure to correctly interpret the depiction.
Preferably the method is designed in such a way that the at least one additional virtual plane and the virtual road plane are depicted in such a perspective correlation to each other and/or to the vehicle model that the perspective correlation corresponds essentially to the perspective correlation in the real vehicle surroundings, in particular in relation to a predetermined viewing point. In this way the occupant of the vehicle can assign the views with respect to certain areas of the vehicle's surroundings, which are imaged on the virtual planes, to the real spatial realities by means of perspectives that are easy to recognize and interpret, and from which the vehicle is depicted.
In another preferred embodiment of the method according to the invention, the perspective in which the virtual road plane and/or the at least one additional virtual plane are displayed is changed as a function of the current odometric data of the vehicle (for example, the speed, the steering angle, etc. of the vehicle) and/or as a function of the recognized driving intention of the driver. The change can relate to the perspective of all virtual planes. In this case the correlation between the individual perspectives among each other is essentially retained. The corresponding necessary adaptation of the perspectives of the virtual road plane and, if desired, other virtual planes can be achieved by changing the shape and/or orientation of the corresponding planes, along with the concomitant change of the imaging function of the respective image to the corresponding plane.
For this purpose, the driving intention of the driver can also be recognized by means of the evaluation of the vehicle data, such as the currently engaged gear or the position of the gear shift setting (e.g., reverse gear, forward gear, parking position) and/or by means of the automatic evaluation of the driver's behavior. The evaluation of the driver's behavior can include, for example, the operating action last performed by the driver or special combinations of operating actions. As an alternative or in addition, the driving intention of the driver can also be recognized by the variation in his viewing direction, such as by means of eye tracking.
In another, especially preferred embodiment of the invention, the generated depiction is changed as a function of the traffic situation, which is recorded or detected preferably by means in the vehicle, such as with the aid of a navigation system or by means of an automatic sensor-based evaluation of the situation. A predetermined event can be, for example, the appearance of an obstacle in the surroundings of the vehicle. In this case the obstacle is detected by means of an imaging device, in particular, by means of a surround sensing system that captures the topography of the surroundings. Furthermore, predetermined events can comprise the wireless reception of signals of road users or other objects.
Conceivable is, for example, a change in the generated depiction as a function of the known and/or automatically determined sources of danger or other road users. The necessary previous knowledge of known sources of danger can come, for example, from an electronic road map. It can also be acquired in a wireless manner from a service provider. The presence and optionally the exact position and/or other data of road users can be made known to the vehicle, for example, by means of vehicle-to-vehicle communication and/or vehicle-to-infrastructure communication.
An object in the surroundings of a vehicle can also be detected, because the vehicle has a radio frequency identification (RFID) chip or a similarly detectable feature. Such an object that is identified on an RFID basis can be depicted, for example, in a camera image or in an image that is artificially generated on the basis of an ultrasonic-based detection of the surroundings.
If another road user, in particular another vehicle and/or a pedestrian, is provided with an RFID chip or other wireless communication means, then this road user can also be detected and localized on the basis of a detection of this chip by the motor vehicle.
Thus, the virtual viewing direction can be changed automatically, for example, in the direction of the relevant object. In this case the depiction can be automatically adjusted, for example, as a function of a hazardous potential, in particular, the hazardous potential of a collision that can be attached to an object (for example, a road user in the current driving situation) in such a way that the generated depiction includes this object (for example, the road user) and/or shows the object from a different perspective. The perspective can be selected in such a way, for example, that the visibility of the object and/or the driver's orientation option is improved.
In addition to other road users, other objects and states of the vehicle surroundings can also be automatically detected on the basis of the acquisition of information from the RFID chip, and can cause the depiction according to embodiments of the invention to change automatically. Conceivable is, for example, an RFID based detection and, if desired, localization of traffic signs, intersections, junctions, entrances, street lights and parking opportunities. Even as a function of such objects and their arrangement with respect to the motor vehicle, the virtual viewing angle of the depiction can be adjusted in a situation-dependent manner, so that the perspective correlation between the individually depicted regions of the vehicle's surroundings stays the same.
The RFID detection can also concern objects that do not directly relate to the driving action, but rather serve commercial purposes and/or belong to the general infrastructure, such as mailboxes, free parking spaces, parking ticket machines, restaurants, vending machines, etc. The generated depiction can also be adjusted as a function of the situation in such a way that such objects are shown to the driver from, for example, a previous perspective. Thus, an object, such as a free parking space, can be shown to the driver in the vehicle's surroundings, especially in a three-dimensional relationship with the model of his vehicle that is depicted in perspective.
In addition to adapting the generated depiction subject to the influence of an RFID chip, at least one information item that is read out of the respective RFID chip and relates to the object, provided with the RFID chip, or its current state can also be displayed by means of the display unit provided according to embodiments of the invention. Instead of the display of the information by the display unit, it is self-evident that the display of the information can also be induced by an additional information communicating unit. The generated depiction can be adjusted, as aforementioned, in such a way that the driver is shown a free parking space in the depicted segment of the surroundings. In addition, it is possible to show the parking fees for the use of the parking space (for example, superimposed on the image of the segment of the surroundings or at another point in the vehicle interior) as additional information read out of the RFID chip.
One embodiment of the invention changes the generated depiction in such a way that the virtual viewing angle generated by the depiction with respect to the virtual road plane and/or the at least one additional virtual plane is changed, but the perspective relations of the images contained in the depiction remain unaffected. To this end, an especially preferred embodiment changes the generated depiction in such a way that the perspective display of the vehicle's surroundings is rotated and/or tilted, in particular about an axis that runs vertically to the virtual road plane. This process makes it especially easy to depict a view of the vehicle's surroundings that corresponds to a movement around the vehicle that is simulated, for example, from a height that equals the size of a human.
Furthermore, an additional, especially preferred embodiment of the invention designs the generated depiction in such a way that a binocularly perceived depth effect in the depiction of the virtual planes is generated. In particular, each portion of the road plane and/or optionally the additional virtual planes is assigned the perceivable depth that corresponds to the above-defined criteria by means of the perspective orientation. The display of a depiction determined in such a way can be implemented, for example, by means of a 3D capable display. In particular, the regions of the vehicle's surroundings that are captured only two-dimensionally are depicted on the virtual planes that are displayed in a spatial position that corresponds to a perspective that is to be depicted according embodiments of to the invention (in particular, relative to another virtual plane). This process offers the advantage of a three-dimensional and thus easy-to-understand and aesthetically advantageous depiction, without any need for the recording means to capture necessarily three-dimensional images, such as stereoscopic images.
As an alternative or in addition, the three-dimensional effect of the individual virtual areas and/or the vehicle model can also be depicted by an adaptive control of the image sharpness as a function of the image region inside at least one virtual plane. The portions of the virtual planes that are to be perceived as significantly nearer or deeper than a certain image depth are imaged, according to embodiments of the invention, less clearly than those image portions that correspond to this specific image depth. The human visual system has the ability to use these differences in the image sharpness to perform a three-dimensional interpretation of the scene. Embodiments of the invention provide that the adaptive control of the image sharpness of individual regions of the image planes can be used to highlight the perspective orientation of the corresponding images. The control of the image sharpness of the individual image regions can be achieved, for example, with the use of known digital sharpness filters, soft focus filters, or optical effects of a recording means.
In another embodiment of the method according to the invention, the virtual road plane is formed as a function of the three-dimensional shape of the road sections imaged thereon, and/or of the objects located thereon. In this way the sources of danger can be emphasized in a way that is especially simple and easy for the driver to understand. The contouring of the road planes can be visualized or highlighted, for example, by the deformation of a lattice structure superimposed on these planes, and/or by an additional geometric transformation on the virtual road plane, including the image imaged thereon.
Another embodiment adds, if desired, to the generated depiction one or more warning indicators that comprise the marking of a potential collision point of one or more parts of the vehicle, such as doors and/or flaps of the vehicle, with one or more objects in the surroundings of the vehicle. The warning indicators can be constructed, for example, as a graphic symbol, such as a three-dimensional action symbol. A warning indicator can be integrated into the depiction in such a way that the spatial reference between the object from which the danger emanates and the potentially endangered point of the vehicle is visualized. A warning indicator can be depicted, for example, at a point between an obstacle and the potentially endangered area of the vehicle (for example, a door of the vehicle) and, for example, at the point of potential damage, caused by inadvertently opening the door.
In an additional, especially preferred embodiment the generated depiction contains one or more trajectories and/or symbols that correspond to the future positions of at least one part of the vehicle. In this case the trajectories are based, in particular, on a pre-calculated or intended direction of movement of the vehicle. This depiction can comprise the depiction of a future position of a part of the vehicle, such as the depiction of the position of a chassis region at a point in time that lies in the near future. For example, the calculated future position of a bumper of the vehicle can be calculated at least at a point in time after, for example, one, two and three seconds. These positions can be calculated in advance in a very simple way assuming, for example, that the driver maintains his current driving style. The positions of the vehicle and/or the parts of the vehicle can be depicted in a way similar to the depiction of the different regions of the vehicle's surroundings.
In the depiction of the intended movement of the vehicle, the results of carrying out an automatic and/or semi-automatic method for moving, in particular for parking a vehicle, can be visualized.
In addition to the described method for informing the occupant of a vehicle, embodiments of the invention additionally comprise a suitable information system comprising an image processing unit, which processes at least two images that are generated by the recording means of the vehicle and contain a variety of road sections of the road plane in the surroundings of the vehicle, in such a manner that the at least two images are displayed on a display unit in the interior of the vehicle. The information system is characterized in that, when working, the image processing unit generates a depiction in which the at least two images are imaged on a virtual road plane, wherein the virtual road plane is displayed in perspective.
In this context the information system can be implemented in that each of the above-described and preferred embodiments of the method according to the invention can be carried out with this system. In addition, embodiments of the invention relate to a vehicle, in particular a motor vehicle, which contains the just-described information system. The image processing unit can be combined with the recording means or with the display unit so as to form a single device.
Preferred embodiments of the invention are described below with reference to the accompanying drawings. The results of this description are additional details, preferred embodiments and further developments of the invention. A plethora of changes and modifications are possible within the scope of the invention without departing from the scope of the invention or its equivalents. In particular, the inventive system or the corresponding method can be designed differently owing to the legal provisions or the country-specific specifications.
The described exemplary embodiments, embodiments and further developments can be used both individually and also in any combination with each other (unless otherwise stated and/or technically infeasible) and are advantageous.
Other objects, advantages and novel features of the present invention will become apparent from the following detailed description of one or more preferred embodiments when considered in conjunction with the accompanying drawings.